Tail Probabilities Research Articles

Contrast estimation of the Vasicek integrated diffusion process for high-frequency data

The purpose of this paper is to study the parameter estimation of the Vasicek integrated diffusion process. Based on the contrast function, the parameter contrast estimators of the Vasicek integrated diffusion process are given, and the mean-square errors of drift parameter estimators are derived, so as to obtain the optimal sampling intervals for the parameter estimators. Then we prove the strong consistency of the contrast estimators by the tail probability exponential inequality of α-mixing long-span high-frequency data. In numerical simulation, we investigate the fitting effect of the estimates under different sampling intervals and different sample sizes. The results indicate that using optimal interval sampling can achieve good estimation performance. We use the daily closing price data of CSI 300 index to conduct an empirical analysis and give a logarithmic price prediction model. The analysis results show that the prediction model works well.

On the empirical approximation to quantiles from Lugannani–Rice saddlepoint formula

Lugannani–Rice saddlepoint formula approximates the tail probability and the cumulative distribution function of the sample mean of independent and equally distributed variables. This note revisits Lugannani–Rice formula with a proposal for inverting it to approximate the quantile of the distribution empirically. The asymptotic behavior of the empirical approximation is assessed theoretically and its numerical accuracy for finite samples is studied and compared with the normal approximation and the second order Cornish-Fisher expansion by means of a simulation study. The outcomes of the simulation experiment shed light on the limitations of the empirical inversions of saddlepoint formulae to approximate quantiles.

Exponential Tail Bounds on Queues: A Confluence of Non- Asymptotic Heavy Traffic and Large Deviations

In general, obtaining the exact steady-state distribution of queue lengths is not feasible. Therefore, we focus on establishing bounds for the tail probabilities of queue lengths. We examine queueing systems under Heavy Traffic (HT) conditions and provide exponentially decaying bounds for the probability P(∈q > x), where ∈ is the HT parameter denoting how far the load is from the maximum allowed load. Our bounds are not limited to asymptotic cases and are applicable even for finite values of ∈, and they get sharper as ∈ - 0. Consequently, we derive non-asymptotic convergence rates for the tail probabilities. Furthermore, our results offer bounds on the exponential rate of decay of the tail, given by -1/2 log P(∈q > x) for any finite value of x. These can be interpreted as non-asymptotic versions of Large Deviation (LD) results. To obtain our results, we use an exponential Lyapunov function to bind the moment-generating function of queue lengths and apply Markov's inequality. We demonstrate our approach by presenting tail bounds for a continuous time Join-the-shortest queue (JSQ) system.

Heterogeneity, climate change, and crop yield distributions: Solvency implications for publicly subsidized crop insurance programs

AbstractClimate change continues to fuel concern about the future cost of publicly subsidized crop insurance programs in developed nations. These changes in climate are expected to alter the upper and lower tails of crop yield distributions differently. This may best be captured by modeling the climate–yield relationship heterogeneously across different parts of the yield distribution. To this end, we consider a mixture model with the parameters expressed as nonparametric functions (to capture any nonlinearities) of weather variables estimated by machine learning methods (neural net). By doing so, we are able to identify possibly heterogeneous effects of climate change on each component, the mixing probabilities, and thus all moments of the yield distribution. We find changing climate alters, quite significantly, the entire shape of the yield distribution. The overall probability of the lower tail tends to increase as temperatures rise, to the point where some yield distributions become positively skewed. Across a range of climate change scenarios, premium rates for fixed guarantees are expected to rise 20–66% relative to no climate change by 2040. However, if we allow the yield guarantees to also fall because of additional losses from climate change, premium rates (albeit not comparable given yield guarantees are different) increase notably less (6–14%), suggesting less solvency issues than first thought.

Stochastic ordering for hitting times of fractional Brownian motions

In this note we prove that, under a suitable transformation, the hitting times of fractional Brownian motions (fBMs) can be ordered stochastically. As a consequence, we derive upper and lower bounds for their cumulative distribution function. We also study some properties, such as continuity and monotonicity, of the probability tail and of the expectation of the supremum of fBMs seen as functions of the Hurst index H. We apply these properties to provide alternative proofs of some results known for fBM.

Measuring Discrete Risks on Infinite Domains: Theoretical Foundations, Conditional Five Number Summaries, and Data Analyses

To accommodate numerous practical scenarios, in this article we extend statistical inference for smoothed quantile estimators from finite domains to infinite domains. We accomplish the task with the help of a newly designed truncation methodology for discrete loss distributions with infinite domains. A simulation study illustrates the methodology in the case of several distributions, such as Poisson, negative binomial, and their zero-inflated versions, which are commonly used in the insurance industry to model claim frequencies. Additionally, we propose a very flexible bootstrap-based approach for use in practice. Using automobile accident data and their modifications, we compute what we have termed the conditional five number summary (C5NS) for the tail risk and construct confidence intervals for each of the five quantiles making up C5NS and then calculate the tail probabilities. The results show that the smoothed quantile approach classifies the tail riskiness of portfolios not only more accurately but also produces lower coefficients of variation in the estimation of tail probabilities than those obtained using the linear interpolation approach.

SCOP: Schrödinger Control Optimal Planning for Goal-based Wealth Management

We consider the problem of optimization of contributions of a financial planner such as a working individual towards a financial goal such as retirement. The objective of the planner is to find an optimal and feasible schedule of periodic installments to an investment portfolio set up towards the goal. Because portfolio returns are random, the practical version of the problem amounts to finding an optimal contribution scheme such that the goal is satisfied at a given confidence level. This paper suggests a semi-analytical approach to a continuous-time version of this problem based on a controlled backward Kolmogorov equation (BKE) which describes the tail probability of the terminal wealth given a contribution policy. The controlled BKE is solved semi-analytically by reducing it to a controlled Schrödinger equation and solving the latter using an algebraic method. Numerically, our approach amounts to finding semi-analytical solutions simultaneously for all values of control parameters on a small grid, and then using the standard two-dimensional spline interpolation to simultaneously represent all satisficing solutions of the original plan optimization problem. Rather than being a point in the space of control variables, satisficing solutions form continuous contour lines (efficient frontiers) in this space.

Stochastic modelling of in-structure concrete strength based on stochastic damage model and Burr distribution

The probability measure of low-quality concrete is essential for the reliability analysis of concrete structures. However, this problem is usually neglected, and the normal distribution or lognormal distribution is often selected as the probability distribution of concrete strength. In this study, a better solution for this problem is given by theoretically deriving of the Burr distribution based on the stochastic damage model. A large amount of in-situ test data in engineering structures is applied to perform a K-S test of different distribution types and to fit the distribution parameters. As a result, the advantage of Burr distribution in representing the tail probability is explained by both theoretical derivation and fitting results. And the Burr distribution is accepted by the K-S test in every strength grade while the other distribution types are all partly rejected.

An explainable unsupervised risk early warning framework based on the empirical cumulative distribution function: Application to dairy safety

Efficient food safety risk assessment significantly affects food safety supervision. However, food detection data of different types and batches show different feature distributions, resulting in unstable detection results of most risk assessment models, lack of interpretability of risk classification, and insufficient risk traceability. This study aims to explore an efficient food safety risk assessment model that takes into account robustness, interpretability and traceability. Therefore, the Explainable unsupervised risk Warning Framework based on the Empirical cumulative Distribution function (EWFED) was proposed. Firstly, the detection data’s underlying distribution is estimated as non-parametric by calculating each testing indicator’s empirical cumulative distribution. Next, the tail probabilities of each testing indicator are estimated based on these distributions and summarized to obtain the sample risk value. Finally, the “3σ Rule” is used to achieve explainable risk classification of qualified samples, and the reasons for unqualified samples are tracked according to the risk score of each testing indicator. The experiments of the EWFED model on two types of dairy product detection data in actual application scenarios have verified its effectiveness, achieving interpretable risk division and risk tracing of unqualified samples. Therefore, this study provides a more robust and systematic food safety risk assessment method to promote precise management and control of food safety risks effectively.

Unsupervised Anomaly Detection of Intermittent Demand for Spare Parts Based on Dual-Tailed Probability

The quick development of machine learning techniques provides a superior capability for manufacturing enterprises to make effective decisions about inventory management based on spare parts demand (SPD) data. Since SPD sequences in practical maintenance applications usually show an intermittent distribution, it is not easy to represent the demand pattern of such sequences. Meanwhile, there are some aspects like manual report errors, environmental interference, sudden project changes, etc., that bring large and unexpected fluctuations to SPD sequences, i.e., anomalous demands. The inventory decision made based on the SPD sequences with anomalous demands is not trusted by enterprise engineers. For such SPD data, there are two great concerns, i.e., false alarms in which sparse demands are recognized to be anomalous and missing alarms in which the anomalous demands are categorized as normal due to their adjacent demands having extreme values. To address these concerns, a new unsupervised anomaly-detection method for intermittent time series is proposed based on a dual-tailed probability. First, the multi-way delay embedding transform (MDT) was applied on the raw SPD sequences to obtain higher-order tensors. Through Tucker tensor decomposition, the disturbance of extreme demands can be effectively reduced. For the reconstructed SPD sequences, then, the tail probability at each time point, as well as the empirical cumulative distribution function were calculated based on the probability of the demand occurrence. Second, to lessen the disturbance of sparse demand, the non-zero demand sequence was distilled from the raw SPD sequence, with the tail probability at each time point being calculated. Finally, the obtained dual-tailed probabilities were fused to determine the anomalous degree of each demand. The proposed method was validated on the two actual SPD datasets, which were collected from a large engineering manufacturing enterprise and a large vehicle manufacturing enterprise in China, respectively. The results demonstrated that the proposed method can effectively lower the false alarm rate and missing alarm rate with no supervised information provided. The detection results were trustworthy enough and, more importantly, computationally inexpensive, showing significant applicability to large-scale after-sales parts management.

ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions

Outlier detection refers to the identification of data points that deviate from a general data distribution. Existing unsupervised approaches often suffer from high computational cost, complex hyperparameter tuning, and limited interpretability, especially when working with large, high-dimensional datasets. To address these issues, we present a simple yet effective algorithm called ECOD (Empirical-Cumulative-distribution-based Outlier Detection), which is inspired by the fact that outliers are often the "rare events" that appear in the tails of a distribution. In a nutshell, ECOD first estimates the underlying distribution of the input data in a nonparametric fashion by computing the empirical cumulative distribution per dimension of the data. ECOD then uses these empirical distributions to estimate tail probabilities per dimension for each data point. Finally, ECOD computes an outlier score of each data point by aggregating estimated tail probabilities across dimensions. Our contributions are as follows: (1) we propose a novel outlier detection method called ECOD, which is both parameter-free and easy to interpret; (2) we perform extensive experiments on 30 benchmark datasets, where we find that ECOD outperforms 11 state-of-the-art baselines in terms of accuracy, efficiency, and scalability; and (3) we release an easy-to-use and scalable (with distributed support) Python implementation for accessibility and reproducibility.

F-Divergences and Their Applications in Lossy Compression and Bounding Generalization Error

In this paper, we provide three applications for f-divergences: (i) we introduce Sanov’s upper bound on the tail probability of the sum of independent random variables based on super-modular f-divergence and show that our generalized Sanov’s bound strictly improves over ordinary one, (ii) we consider the lossy compression problem which studies the set of achievable rates for a given distortion and code length. We extend the rate-distortion function using mutual f-information and provide new and strictly better bounds on achievable rates in the finite blocklength regime using super-modular f-divergences, and (iii) we provide a connection between the generalization error of algorithms with bounded input/output mutual f-information and a generalized rate-distortion problem. This connection allows us to bound the generalization error of learning algorithms using lower bounds on the f-rate-distortion function. Our bound is based on a new lower bound on the rate-distortion function that (for some examples) strictly improves over previously best-known bounds.

Strong consistency of parameter estimation for the CIR integrated diffusion process with long-span high-frequency data

In this article, we utilize the contrast function of the integrated diffusion process to derive parameter contrast estimators for the Cox-Ingersoll-Ross integrated diffusion process and discuss the asymptotic properties of the estimators using the mixing properties of the process. We first give the tail probability exponential inequality of the mixing long-span high-frequency data. Then, applying this inequality, we prove the strong consistency of the contrast estimators for the drift parameters and diffusion parameter of the CIR integrated diffusion process. In order to obtain these asymptotic properties, we require relatively low order r of the negative power moment of the diffusion process. In the existing literature, r > 9 is required in order to obtain weak consistency of the estimators. However, in our theorems, only r > 2 is required for the strong consistency of the diffusion parameter estimator, while r > 4 for the strong consistency of the drift parameter estimators. In the simulation analysis, we use the Euler discretization algorithm and the Milstein discretization algorithm, and the results show that the optimal sampling interval is around . We use the daily log price data of the CSI 300 Index to estimate the CIR integrated diffusion model and construct a Bootstrap prediction model for the log price. The results show that the model has a good predictive performance.

Network homophily via tail inequalities.

Homophily is the principle whereby "similarity breeds connections." We give a quantitative formulation of this principle within networks. Given a network and a labeled partition of its vertices, the vector indexed by each class of the partition, whose entries are the number of edges of the subgraphs induced by the corresponding classes, is viewed as the observed outcome of the random vector described by picking labeled partitions at random among labeled partitions whose classes have the same cardinalities as the given one. This is the recently introduced random coloring model for network homophily. In this perspective, the value of any homophily score Θ, namely, a nondecreasing real-valued function in the sizes of subgraphs induced by the classes of the partition, evaluated at the observed outcome, can be thought of as the observed value of a random variable. Consequently, according to the score Θ, the input network is homophillic at the significance level α whenever the one-sided tail probability of observing a value of Θ at least as extreme as the observed one is smaller than α. Since, as we show, even approximating α is an NP-hard problem, we resort to classical tails inequality to bound α from above. These upper bounds, obtained by specializing Θ, yield a class of quantifiers of network homophily. Computing the upper bounds requires the knowledge of the covariance matrix of the random vector, which was not previously known within the random coloring model. In this paper we close this gap. Interestingly, the matrix depends on the input partition only through the cardinalities of its classes and depends on the network only through its degrees. Furthermore all the covariances have the same sign, and this sign is a graph invariant. Plugging this structure into the bounds yields a meaningful, easy to compute class of indices for measuring network homophily. As demonstrated in real-world network applications, these indices are effective and reliable, and may lead to discoveries that cannot be captured by the current state of the art.

A multiobjective optimization approach for threshold determination in extreme value analysis for financial time series

The literature on the extreme value theory threshold optimization problem for multiple time series analysis does not consider determining a single optimal tail probability for all marginal distributions. With multiple tail probabilities, their discrepancy results in a differing number of exceedances, which may favour a particular marginal series. In this study, we propose a single optimal tail probability by integrating trade-offs among multiple time series within an MOO framework. Mathematically, our approach links the peaks-over-threshold technique and goal programming technique by developing a set of regression functions, which represent continuous paths of possible tail areas for multiple time series, and we formulate them at the desired levels within a multiobjective optimization framework. The optimal solution is found as the minimum Chebyshev variant weighted value. Our approach advances the development of the peaks-over-threshold method by considering the characteristics of a group of time series collectively instead of independently. The proposed optimal tail probability can be considered an optimal reference point for practical risk investment portfolio analysis that employs an identical tail size across multiple time series data. The daily log returns of four U.S. stock market indices, namely, S&P 500, NASDAQ Composite, NYSE Composite, and Russell 2000, from 1 July 1992 to 30 June 2022 are studied empirically.

On Underdispersed Count Kernels for Smoothing Probability Mass Functions

Only a few count smoothers are available for the widespread use of discrete associated kernel estimators, and their constructions lack systematic approaches. This paper proposes the mean dispersion technique for building count kernels. It is only applicable to count distributions that exhibit the underdispersion property, which ensures the convergence of the corresponding estimators. In addition to the well-known binomial and recent CoM-Poisson kernels, we introduce two new ones such the double Poisson and gamma-count kernels. Despite the challenging problem of obtaining explicit expressions, these kernels effectively smooth densities. Their good performances are pointed out from both numerical and comparative analyses, particularly for small and moderate sample sizes. The optimal tuning parameter is here investigated by integrated squared errors. Also, the added advantage of faster computation times is really very interesting. Thus, the overall accuracy of two newly suggested kernels appears to be between the two old ones. Finally, an application including a tail probability estimation on a real count data and some concluding remarks are given.

Model misspecification of Log-Normal and Birnbaum-Saunders distributions

Model misspecification can be a serious issue in any lifetime data analysis. Assumption of the correct model is very important particularly in prediction for future observations and for estimating the tail probabilities of any lifetime distribution. In this paper we have considered the model misspecification of the log-normal and Birnbaum-Saunders distributions. These two distributions have striking similarities both in terms of the probability density functions and hazard functions, in certain range of parameter values, which makes it extremely difficult to detect the correct model. Hence, to identify the correct model, first the conventional method of ratio of maximized likelihood approach has been tried. The necessary theoretical results have been derived. Based on extensive simulations it has been observed that for certain range of parameter values the model misspecification can be quite high even for very large sample sizes. Some of the other methods like the ratio of maximized product of spacings and minimized Kolmogorov-Smirnov distance, have also been explored. But none of the method performs uniformly better than the other over the entire range of the parameter space. Some counter intuitive results have also been obtained. Hence, we finally conclude that it is extremely difficult to choose the correct model in case of log-normal and Birnbaum-Saunders distribution for certain range of parameter values. Finally, we have suggested a modified ratio of maximized likelihood approach, and the performances are quite satisfactory.

Exponential Tail Bounds on Queues

A popular approach to computing performance measures of queueing systems (such as delay and queue length) is studying the system in an asymptotic regime. However, these results are only valid in the limit and often provide bounds for the pre-limit systems that are not optimized and, hence, give loose bounds for the tail probabilities. In this paper, we provide optimized bounds for the tail probabilities of the scaled total queue length in a load-balancing system under Join the Shortest Queue (JSQ). Our bounds characterize the rate of convergence of the tail probabilities to the corresponding heavy traffic values. For the tail probability of the JSQ system, our bounds yield a multiplicative error that arises from three factors: pre-limit tail, pre-exponent error, and State-Space Collapse (SSC). As an immediate corollary of our main theorem, we provide a bound to the tail probabilities of a single-server queue. In this case, the multiplicative error only consists of pre-limit tail and pre-exponent error, since there is no state-space collapse.

An exact bound for tail probabilities for a class of conditionally symmetric bounded martingales

We consider the class, say ℳn,sym, of martingales Mn = X1 + ⋯ + Xn with conditionally symmetric bounded differences Xk such that |Xk | ≤ 1. We find explicitly a solution, say Dn(x), of the variational problem Dn(x) ≝ sup Mn ∈ℳn,sym ℙ {Mn ≥ x}. We show that this problem is equivalent to one when you want to find out the symmetric random walk with bounded length of steps which maximizes the probability to visit an interval [x;∞]. The function x \mapsto Dn(x) allows a simple description and is closely related to the binomial tail probabilities. We can interpret the result as a final and optimal upper bound ℙ{Mn ≥ x} ≤ Dn(x), x ∈ ℝ, for the tail probability ℙ {Mn ≥ x}.

Sharper Concentration Inequalities for Median-of-Mean Processes

The Median-of-Mean (MoM) estimation is an efficient statistical method for handling data with contamination. In this paper, we propose a variance-dependent MoM estimation method using the tail probability of a binomial distribution. The bound of this method is better than the classical Hoeffding method under mild conditions. This method is then used to study the concentration of variance-dependent MoM empirical processes and sub-Gaussian intrinsic moment norm. Finally, we give the bound of the variance-dependent MoM estimator with distribution-free contaminated data.